Repository navigation
Sign each Claude Code turn's usage as it happens; tips prefers signed counts (#865) - #871
Merged
Merged
Conversation
…rs signed counts (#865) The transcript tailer already signs every assistant line as a model.exchange (model_event). It now also signs the message's usage (input, output, cache-read and cache-write tokens, normalised as the LLM gateway records them) once per message id in the run: on the message's last line a read takes in, and only at its final counts (a line with a stop_reason). The ids signed go into the dev-mode progress log, so a restarted tailer never signs one again; a subagent transcript's copy of the message that started it is not signed twice. No new event type, schema or RPC change: model.exchange and model_event already carry `usage`, and model_event never feeds the provenance index. tracekit tips joins a signed step to its transcript message by message id (as well as tool_use id and counts), takes the signed counts in the transcript's place (keeping the transcript's cache-lifetime split while it adds up), counts an exchange id once, and says where each figure's counts came from: the tip's `from`, the context tip's arithmetic line, each wait's `src`. Docs: privacy.md rows for the v2 gateway and tailer fields, a limits.md row (signed at read time, so an edit before the read isn't caught; the gateway signs off the wire), and the TIPS sources paragraph. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…eway; gateway and tailer count once (review of #871) - A gateway's record of a turn without a tool call joins its transcript message by its counts even where the tailer signed it too, so the turn counts once; where both signed it, the gateway's counts (signed off the wire) are kept. The count pass compares with the counts the tailer signed. - A record joins a transcript turn by message id only from a claude-code run whose records name a tool call of that session, signed from the turn's first line to MATCH_S after its last; by a tool_use id only from an LLM gateway's record (its source). Any other record of a transcript's turn is refused: it counts for nothing, the transcript's counts stand, and `tracekit tips` and the TIPS page say how many. An exchange id counts once in a run, and once across runs where no transcript holds it. - The tailer signs a turn that stopped for a tool use on a tool_use line, so the viewer's model line names its calls when a read ends before them. - The viewer's token totals count an exchange id once. - Docs: the limits rows (made-up lines, read only at the next hook event after a kill or park, the 25 MB check, the session's own run token), the privacy rows (tool use ids and names, tool_results_sent, what output counts reveal), and the TIPS matching paragraph. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…t; the viewer counts a gateway-and-tailer turn once (2nd review of #871) - In tips' join, a transcript turn takes its records' counts in a pass only when they all agree; when they don't, each is refused and the turn keeps the counts it had. A run that names one call of the session can push a signed turn down to the transcript's counts, but no record's counts go over another's. The gateway's counts are still kept over the tailer's. - An exchange id no transcript holds that two runs signed counts once when their counts agree, and for neither (refused) when they don't. - The viewer's TOKENS IN and OUT join a turn the LLM gateway and the transcript tailer both signed (by exchange id, tool_use id, or the same counts within 5 minutes) and keep the gateway's counts. - Docs: the limits row says any run that names a call of the session can sign for its turns, and that conflicting records drop to the transcript's counts; the TIPS paragraph says the same. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
… join takes one turn per record (final check of #871) - A run could hide spend an LLM gateway signed: by signing the gateway's exchange ids at other counts (both were refused), or by moving a turn in the transcript so the gateway's record fell outside its window (refused). A gateway's record now always counts: on a conflict only the other records are refused, one that joins no turn in its window counts on its own, and two gateways that disagree on a turn both count. A run can make tips count more, never less. - The viewer joins records by their counts one turn per record: two turns with the same counts that the gateway and the tailer both signed count as two, not three. - Docs: the limits row and its remedy, and the TIPS paragraph. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
The agent's user could write transcript lines under the gateway's exchange ids (the dev store shows them): pass 1 took the gateway's records for the tailer's, refused them, and only the made-up turns counted. A gateway's record now skips the message-id pass, so it joins by tool_use id or counts, or counts on its own. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
LoopGlitch26
added a commit
that referenced
this pull request
Nov 15, 2026
…the lessons part Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
10 of 11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #865.
What changes
model.exchangeresponse (model_event, exchange id = the message id). It now addsusage(input, output, cache-read and cache-write tokens, normalised byusage.from_anthropic, as the LLM gateway records them) once per message id in the run, only at the turn's final counts (a line with astop_reason), on the message's last line a read takes in, and for a turn that stopped for a tool, on a tool_use line. The ids it signed go into the dev-mode progress log ([transcript, offset, digest, message id]), so a restarted tailer signs none again, and a subagent transcript that opens with a copy of the message that started it doesn't sign it twice.tracekit tipsprefers signed usage, from the right record only. A signed record takes a transcript turn's place, with its signed counts (the transcript keeps which agent took the turn, when, and the cache-lifetime split while it adds up), in three passes:Where the gateway and the tailer both signed a turn, it counts once and the gateway's counts are kept (signed off the wire). Other records of one turn that disagree put none of their counts on it: each is refused and the turn keeps the counts it had, so a run can push a signed turn down to the transcript's counts but never put its own over another's. A gateway's record never counts for nothing: it never joins by message id (a transcript line can't claim its exchange id), on a conflict only the other records are refused, one that joins no turn in its window counts on its own, and two gateways that disagree both count, so a run can make tips count more, never hide what a gateway signed. Any other record of a transcript's turn is refused too: it counts for nothing, the transcript's counts stand, and
sources.signed.refused,tracekit tipsand the TIPS page say how many. An exchange id counts once in a run; across runs where no transcript holds it, once when the counts agree; when they don't, a gateway's record counts and the others are refused, and without one none counts. Each tip says where its counts came from:from, a line in the context tip's arithmetic, andsrcon each cache wait.docs/privacy.mdrows for the v2 gateway and tailer fields (usage, model, message id, tool use ids and names,tool_results_sentand the signed time in clear; the line only as a salted commitment, in the vault only when the signer keeps content) and what output counts reveal;docs/limits.mdrows (signed at read time, made-up lines, the 25 MB check, the session's own run token) and the TIPS row; the TIPS paragraph indocs/viewer.md; one sentence indocs/coding-agents.md.Design choice
The smallest path used what exists.
model.exchangealready has ausagefield,model_eventalready takes it (RPC v23 unchanged), and the signer already writes it. So: no new event type, no schema change, no RPC contract change, nothing for the verifiers or the conformance kit.model_eventnever feeds the provenance index (onlycompleteof untrusted tools andobservedo), so usage numbers can't become trusted values there.The issue pictured the hook reading new transcript bytes at each event. The v2 hook already starts a detached tailer per run that does exactly that (from an offset kept beside the session state, checked, with signed
tailer_lostgaps for a broken transcript), so hook latency doesn't change: PreToolUse stayed at a 73-79 ms median in the user runs, with and without this change.The record names no session: any run can call
model_eventwith any message id and any counts. Tips therefore binds a record to a turn by its run, its time and, for the gateway, the gateway's own identity; it never lets one record's counts go over another's, and never drops a gateway's. What that can't close: any run that names one of the session's calls joins its turns (the agent's user can read the session's run token, and start runs of its own), so it can sign made-up counts for a turn no other run signed within the window (a limits row; the gateway is the answer where spend must not rest on the agent's word).Findings from the user runs:
stop_reason, output 16) and a final line later (output 243). Over 120 days here, 68,076 subagent messages of that build never got a final line (median output 8). Current builds (2.1.308, 2.1.311) write astop_reasonon every line; 17 of 88,119 messages in two weeks had none (interrupted turns). Those stay with the transcript tier.refused); with the lines written live, all 109 joined.Fail-first
Each test that a fix needs, run on the code before that fix: round 1 on main (f32c2fe), the 1st review's on 421d341, the 2nd review's on 2d13c6f, the final check's on 3bf3fca and e7a3e1a.
test_tailer.py::Tailer::test_a_messages_usage_is_signed_once_on_its_last_linetest_tailer.py::Tailer::test_a_turn_streamed_over_several_lines_is_signed_at_its_final_countstest_tailer.py::Tailer::test_a_message_read_over_two_wakes_and_its_subagents_copy_of_it_sign_its_usage_oncetest_tailer.py::Tailer::test_a_turns_usage_is_signed_in_clear_and_its_text_only_as_a_commitmenttest_tailer.py::Tailer::test_usage_never_enters_the_provenance_indextest_tailer.py::HookRestartsTheTailer::test_a_tailer_started_again_signs_no_usage_the_stopped_one_signedtest_tips.py::Sources::test_signed_usage_counts_before_the_transcripts_and_each_tip_says_where_its_counts_came_fromtest_tips.py::Sources::test_a_turn_signed_twice_counts_oncetest_tips.py::Sources::test_a_session_the_gateway_signed_and_the_transcripts_hold_counts_once(changed:fromis signed records)test_tips.py::Sources::test_a_turn_the_gateway_and_the_tailer_both_signed_counts_once_at_the_gateways_countstest_tips.py::Sources::test_only_the_sessions_own_run_signs_its_turns_and_only_after_them(5 subtests)test_tips.py::Page::test_the_read_line_says_what_counted_once_and_that_every_project_is_listed(changed)test_tips.py::Page::test_the_viewers_token_totals_count_a_turn_signed_twice_oncetest_tips.py::Sources::test_records_of_one_turn_that_disagree_put_none_of_their_counts_on_it(forger after, forger first)test_tips.py::Sources::test_an_exchange_id_two_runs_signed_with_other_counts_counts_for_neithertest_tips.py::Page::test_the_viewers_token_totals_count_a_turn_the_gateway_and_the_tailer_signed_oncetest_tips.py::Sources::test_a_gateways_record_never_counts_for_nothing(a bot re-signs the gateway's ids; a moved transcript turn; two gateways disagree)test_tips.py::Sources::test_a_transcript_line_under_a_gateways_exchange_id_leaves_the_gateways_record_counted(the gateway's run namedgateway, orclaude-code)Mutation check
One mutation at a time on the named file, the guarding tests run, the file written back. 31 of 31 killed.
mid not in send.signedstop_reasonuses or stop != "tool_use"conditionobservethe usage as a trusted inputfromnames the agent's source, not its counts'tracekit tipsdoesn't say how many were refusedUser run
The hook run as Claude Code calls it (
python -I -m tracekit.integrations.claude_code EVENT, payload on stdin: SessionStart, UserPromptSubmit, PreToolUse/PostToolUse for the read-only calls, Stop, SessionEnd) against a scratch dev signer (its own HOME, runtime dir andCLAUDE_CONFIG_DIR), with a real transcript copy from~/.claude/projectsappended in 25 chunks as Claude Code would write it and its subagents' transcripts appearing mid-session. The source transcripts were only read; the copies are deleted. Each replay ran on this branch and on main.tracekit verifyexit 0: "VERIFIED: 166 tool calls in 539 signed records"tracekit tips --jsonon the same scratch stores and transcripts:sources.signed.also{runs: 0, steps: 0}{runs: 1, steps: 1509}(first head)from{0, 0}, cache tip from transcriptsalso {runs: 1, steps: 109},refused: 0also {0, 0},refused: 109: signed days after the lines, so the transcript's counts standGate
PATH=<repo>/.venv/bin:$PATH PYTHONPATH=$PWD make test-ts lint build testin the worktree, on e7a3e1a: exit 0. TS 173 tests (161 pass, 12 skipped, 0 fail); pyflakes clean; sdist and wheel built,twine checkpassed; pytest 5332 passed, 447 skipped, 10 xfailed, 74,947 subtests passed (35 min). On this head (one line intips._joinand its test):make lintclean, andtests/test_tips.py,test_tailer.py,test_view.py,test_view_ui.py,test_docs.py: 434 passed, 1 skipped, 325 subtests passed. macOS only: CI's Linux run is the merge gate.Limits and open points
docs/limits.md).stop_reasonline is left unsigned (interrupted turns; older builds' streamed subagent lines).--data-dir) a session's subagent turns count as its main session's (lean:intips.signed_steps).